{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Pandas 处理异常值",
        "",
        "本教程详细介绍如何在 Pandas 中检测和处理异常值（离群值）。",
        "",
        "## 目录",
        "1. 异常值检测方法",
        "2. 使用统计方法检测",
        "3. 使用可视化方法检测",
        "4. 异常值处理方法",
        "5. 高级技巧"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 导入库"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import pandas as pd",
        "import numpy as np",
        "import matplotlib.pyplot as plt",
        "import seaborn as sns"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 1. 异常值检测方法",
        "",
        "### 方法说明",
        "",
        "异常值（Outlier）是指与数据集中其他观测值显著不同的值。",
        "",
        "**主要检测方法：**",
        "1. **统计方法**：IQR方法、Z-score方法、标准差方法",
        "2. **可视化方法**：箱线图、散点图、直方图",
        "3. **机器学习方法**：Isolation Forest、DBSCAN等",
        "",
        "**特点：**",
        "- ✅ 统计方法简单快速",
        "- ✅ 可视化方法直观",
        "- ✅ 需要结合业务理解判断",
        "",
        "**适用场景：** 数据探索阶段，发现异常数据"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例1: 创建包含异常值的数据"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "np.random.seed(42)",
        "# 创建正常数据",
        "normal_data = np.random.normal(50, 10, 95)  # 均值50，标准差10，95个值",
        "",
        "# 添加异常值",
        "outliers = np.array([10, 150, 5, 200])",
        "",
        "# 合并数据",
        "all_data = np.concatenate([normal_data, outliers])",
        "",
        "df = pd.DataFrame({",
        "    '数值': all_data,",
        "    '类别': ['正常'] * 95 + ['异常'] * 4",
        "})",
        "",
        "print(\"数据统计:\")",
        "print(df['数值'].describe())",
        "print(\"\\n数据形状:\", df.shape)",
        "print(\"\\n异常值数量:\", len(outliers))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 2. 使用统计方法检测",
        "",
        "### 方法1: IQR方法（四分位距法）",
        "",
        "IQR = Q3 - Q1",
        "异常值：< Q1 - 1.5*IQR 或 > Q3 + 1.5*IQR",
        "",
        "**特点：**",
        "- ✅ 对异常值不敏感（基于中位数）",
        "- ✅ 适合非正态分布数据",
        "- ✅ 经典方法，广泛使用",
        "",
        "**适用场景：** 大多数数值型数据"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例1: 使用IQR方法检测异常值"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "def detect_outliers_iqr(data):",
        "    \"\"\"使用IQR方法检测异常值\"\"\"",
        "    Q1 = data.quantile(0.25)",
        "    Q3 = data.quantile(0.75)",
        "    IQR = Q3 - Q1",
        "    ",
        "    lower_bound = Q1 - 1.5 * IQR",
        "    upper_bound = Q3 + 1.5 * IQR",
        "    ",
        "    outliers = (data < lower_bound) | (data > upper_bound)",
        "    ",
        "    return outliers, lower_bound, upper_bound",
        "",
        "# 检测异常值",
        "outliers_mask, lower_bound, upper_bound = detect_outliers_iqr(df['数值'])",
        "",
        "print(\"IQR方法检测结果:\")",
        "print(f\"下界: {lower_bound:.2f}\")",
        "print(f\"上界: {upper_bound:.2f}\")",
        "print(f\"\\n异常值数量: {outliers_mask.sum()}\")",
        "print(\"\\n异常值:\")",
        "print(df[outliers_mask])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 方法2: Z-score方法",
        "",
        "Z-score = (x - μ) / σ",
        "异常值：|Z-score| > 3（或2.5）",
        "",
        "**特点：**",
        "- ✅ 基于均值和标准差",
        "- ✅ 适合正态分布数据",
        "- ⚠️ 对异常值敏感（均值受异常值影响）",
        "",
        "**适用场景：** 正态分布的数据"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例2: 使用Z-score方法检测异常值"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "def detect_outliers_zscore(data, threshold=3):",
        "    \"\"\"使用Z-score方法检测异常值\"\"\"",
        "    z_scores = np.abs((data - data.mean()) / data.std())",
        "    outliers = z_scores > threshold",
        "    return outliers, z_scores",
        "",
        "# 检测异常值",
        "outliers_mask, z_scores = detect_outliers_zscore(df['数值'], threshold=3)",
        "",
        "print(\"Z-score方法检测结果:\")",
        "print(f\"\\n异常值数量: {outliers_mask.sum()}\")",
        "print(\"\\n异常值及其Z-score:\")",
        "outliers_df = df[outliers_mask].copy()",
        "outliers_df['Z_score'] = z_scores[outliers_mask]",
        "print(outliers_df)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 方法3: 标准差方法",
        "",
        "异常值：< μ - 3σ 或 > μ + 3σ",
        "",
        "**特点：**",
        "- ✅ 简单直观",
        "- ✅ 适合正态分布",
        "- ⚠️ 与Z-score方法类似",
        "",
        "**适用场景：** 正态分布数据"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例3: 使用标准差方法检测异常值"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "def detect_outliers_std(data, n_std=3):",
        "    \"\"\"使用标准差方法检测异常值\"\"\"",
        "    mean = data.mean()",
        "    std = data.std()",
        "    ",
        "    lower_bound = mean - n_std * std",
        "    upper_bound = mean + n_std * std",
        "    ",
        "    outliers = (data < lower_bound) | (data > upper_bound)",
        "    ",
        "    return outliers, lower_bound, upper_bound",
        "",
        "# 检测异常值",
        "outliers_mask, lower_bound, upper_bound = detect_outliers_std(df['数值'], n_std=3)",
        "",
        "print(\"标准差方法检测结果:\")",
        "print(f\"均值: {df['数值'].mean():.2f}\")",
        "print(f\"标准差: {df['数值'].std():.2f}\")",
        "print(f\"下界: {lower_bound:.2f}\")",
        "print(f\"上界: {upper_bound:.2f}\")",
        "print(f\"\\n异常值数量: {outliers_mask.sum()}\")",
        "print(\"\\n异常值:\")",
        "print(df[outliers_mask])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 3. 使用可视化方法检测",
        "",
        "可视化是最直观的异常值检测方法。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例1: 箱线图（Box Plot）"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# 创建箱线图",
        "plt.figure(figsize=(8, 6))",
        "plt.boxplot(df['数值'])",
        "plt.title('箱线图检测异常值')",
        "plt.ylabel('数值')",
        "plt.grid(True)",
        "plt.show()",
        "",
        "# 使用pandas绘制",
        "df['数值'].plot(kind='box', title='箱线图检测异常值')",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例2: 散点图"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "plt.figure(figsize=(10, 6))",
        "plt.scatter(range(len(df)), df['数值'], alpha=0.6)",
        "plt.axhline(y=df['数值'].quantile(0.75) + 1.5 * (df['数值'].quantile(0.75) - df['数值'].quantile(0.25)), ",
        "            color='r', linestyle='--', label='上界')",
        "plt.axhline(y=df['数值'].quantile(0.25) - 1.5 * (df['数值'].quantile(0.75) - df['数值'].quantile(0.25)), ",
        "            color='r', linestyle='--', label='下界')",
        "plt.title('散点图检测异常值')",
        "plt.xlabel('索引')",
        "plt.ylabel('数值')",
        "plt.legend()",
        "plt.grid(True)",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例3: 直方图"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "plt.figure(figsize=(10, 6))",
        "plt.hist(df['数值'], bins=30, alpha=0.7, edgecolor='black')",
        "plt.axvline(x=df['数值'].mean(), color='r', linestyle='--', label='均值')",
        "plt.axvline(x=df['数值'].mean() + 3*df['数值'].std(), color='orange', linestyle='--', label='+3σ')",
        "plt.axvline(x=df['数值'].mean() - 3*df['数值'].std(), color='orange', linestyle='--', label='-3σ')",
        "plt.title('直方图检测异常值')",
        "plt.xlabel('数值')",
        "plt.ylabel('频数')",
        "plt.legend()",
        "plt.grid(True)",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 4. 异常值处理方法",
        "",
        "### 方法说明",
        "",
        "检测到异常值后，需要决定如何处理。",
        "",
        "**主要方法：**",
        "1. **删除**：确认是错误数据时删除",
        "2. **替换**：用边界值、均值、中位数替换",
        "3. **变换**：对数变换、平方根变换等",
        "4. **分箱**：将连续变量离散化",
        "5. **保留**：如果是真实极端值，可以保留",
        "",
        "**选择策略：**",
        "- 数据录入错误 → 删除或修正",
        "- 测量误差 → 删除或替换",
        "- 真实极端值 → 保留或变换",
        "- 线性模型 → 必须处理",
        "- 树模型 → 可以保留"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例1: 删除异常值"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# 使用IQR方法检测异常值",
        "outliers_mask, _, _ = detect_outliers_iqr(df['数值'])",
        "",
        "print(\"删除前:\")",
        "print(f\"数据形状: {df.shape}\")",
        "print(f\"异常值数量: {outliers_mask.sum()}\")",
        "",
        "# 删除异常值",
        "df_cleaned = df[~outliers_mask].copy()",
        "",
        "print(\"\\n删除后:\")",
        "print(f\"数据形状: {df_cleaned.shape}\")",
        "print(f\"删除的行数: {len(df) - len(df_cleaned)}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例2: 用边界值替换（Winsorization）"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "def winsorize(data, lower_percentile=5, upper_percentile=95):",
        "    \"\"\"Winsorization：用百分位数替换异常值\"\"\"",
        "    lower_bound = data.quantile(lower_percentile / 100)",
        "    upper_bound = data.quantile(upper_percentile / 100)",
        "    ",
        "    data_winsorized = data.copy()",
        "    data_winsorized[data < lower_bound] = lower_bound",
        "    data_winsorized[data > upper_bound] = upper_bound",
        "    ",
        "    return data_winsorized",
        "",
        "df_winsorized = df.copy()",
        "df_winsorized['数值'] = winsorize(df_winsorized['数值'])",
        "",
        "print(\"Winsorization结果:\")",
        "print(\"原始数据统计:\")",
        "print(df['数值'].describe())",
        "print(\"\\n处理后的数据统计:\")",
        "print(df_winsorized['数值'].describe())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例3: 用统计值替换"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# 使用IQR方法检测异常值",
        "outliers_mask, lower_bound, upper_bound = detect_outliers_iqr(df['数值'])",
        "",
        "# 用中位数替换异常值",
        "df_median = df.copy()",
        "df_median.loc[outliers_mask, '数值'] = df_median['数值'].median()",
        "",
        "print(\"用中位数替换异常值:\")",
        "print(\"原始异常值:\")",
        "print(df[outliers_mask]['数值'].values)",
        "print(\"\\n替换后的值:\")",
        "print(df_median.loc[outliers_mask, '数值'].values)",
        "",
        "# 用均值替换",
        "df_mean = df.copy()",
        "df_mean.loc[outliers_mask, '数值'] = df_mean['数值'].mean()",
        "",
        "print(\"\\n用均值替换异常值:\")",
        "print(df_mean.loc[outliers_mask, '数值'].values)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例4: 数据变换"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# 对数变换",
        "df_log = df.copy()",
        "df_log['数值_对数'] = np.log1p(df_log['数值'])  # log1p = log(1+x)，避免log(0)",
        "",
        "# 平方根变换",
        "df_sqrt = df.copy()",
        "df_sqrt['数值_平方根'] = np.sqrt(df_sqrt['数值'])",
        "",
        "print(\"原始数据统计:\")",
        "print(df['数值'].describe())",
        "print(\"\\n对数变换后统计:\")",
        "print(df_log['数值_对数'].describe())",
        "print(\"\\n平方根变换后统计:\")",
        "print(df_sqrt['数值_平方根'].describe())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 示例5: 分箱（Binning）"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# 将连续变量离散化",
        "df_binned = df.copy()",
        "",
        "# 使用cut进行等宽分箱",
        "df_binned['数值_分箱'] = pd.cut(df_binned['数值'], bins=5, labels=['很低', '低', '中', '高', '很高'])",
        "",
        "print(\"分箱结果:\")",
        "print(df_binned[['数值', '数值_分箱']].head(10))",
        "",
        "# 使用qcut进行等频分箱",
        "df_binned['数值_等频分箱'] = pd.qcut(df_binned['数值'], q=5, labels=['很低', '低', '中', '高', '很高'])",
        "",
        "print(\"\\n等频分箱结果:\")",
        "print(df_binned[['数值', '数值_等频分箱']].head(10))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 5. 高级技巧",
        "",
        "### 技巧1: 标记异常值但不删除"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "df_marked = df.copy()",
        "",
        "# 检测异常值",
        "outliers_mask, _, _ = detect_outliers_iqr(df_marked['数值'])",
        "",
        "# 添加异常值标记",
        "df_marked['是否异常'] = outliers_mask.astype(int)",
        "",
        "print(\"添加异常值标记:\")",
        "print(df_marked[['数值', '是否异常']].head(10))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 技巧2: 按分组检测异常值"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# 创建分组数据",
        "df_group = pd.DataFrame({",
        "    '类别': ['A'] * 50 + ['B'] * 50,",
        "    '数值': list(np.random.normal(50, 10, 50)) + list(np.random.normal(100, 15, 50)) + [200]",
        "})",
        "",
        "print(\"原始数据:\")",
        "print(df_group.head())",
        "",
        "# 按类别分组检测异常值",
        "def detect_outliers_group(group):",
        "    Q1 = group.quantile(0.25)",
        "    Q3 = group.quantile(0.75)",
        "    IQR = Q3 - Q1",
        "    lower_bound = Q1 - 1.5 * IQR",
        "    upper_bound = Q3 + 1.5 * IQR",
        "    return (group < lower_bound) | (group > upper_bound)",
        "",
        "df_group['是否异常'] = df_group.groupby('类别')['数值'].transform(detect_outliers_group)",
        "",
        "print(\"\\n按分组检测异常值:\")",
        "print(df_group[df_group['是否异常']])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 技巧3: 多变量异常值检测"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# 创建多变量数据",
        "df_multi = pd.DataFrame({",
        "    'X': np.random.normal(50, 10, 100),",
        "    'Y': np.random.normal(50, 10, 100)",
        "})",
        "",
        "# 添加异常值",
        "df_multi.loc[95] = [150, 150]",
        "df_multi.loc[96] = [10, 10]",
        "",
        "print(\"多变量数据:\")",
        "print(df_multi.tail())",
        "",
        "# 使用马氏距离检测多变量异常值",
        "from scipy.spatial.distance import mahalanobis",
        "from scipy.stats import chi2",
        "",
        "mean = df_multi[['X', 'Y']].mean()",
        "cov = df_multi[['X', 'Y']].cov()",
        "",
        "# 计算马氏距离",
        "distances = df_multi[['X', 'Y']].apply(",
        "    lambda row: mahalanobis(row, mean, cov), axis=1",
        ")",
        "",
        "# 使用卡方分布判断异常值（95%置信区间）",
        "threshold = np.sqrt(chi2.ppf(0.95, df=2))  # df=2 因为有2个变量",
        "df_multi['是否异常'] = distances > threshold",
        "",
        "print(\"\\n多变量异常值检测结果:\")",
        "print(df_multi[df_multi['是否异常']])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 总结",
        "",
        "### 方法对比",
        "",
        "| 检测方法 | 优点 | 缺点 | 适用场景 |",
        "|---------|------|------|----------|",
        "| IQR方法 | 对异常值不敏感 | 可能遗漏边缘值 | 大多数数值型数据 |",
        "| Z-score | 标准方法 | 受异常值影响 | 正态分布数据 |",
        "| 箱线图 | 直观可视化 | 主观判断 | 探索性分析 |",
        "| 散点图 | 直观可视化 | 主观判断 | 探索性分析 |",
        "",
        "### 处理策略对比",
        "",
        "| 处理策略 | 优点 | 缺点 | 适用场景 |",
        "|---------|------|------|----------|",
        "| 删除 | 简单直接 | 损失数据 | 确认错误数据 |",
        "| Winsorization | 保留数据 | 可能失真 | 真实极端值 |",
        "| 统计值替换 | 保持统计特性 | 可能偏差 | 缺失值处理 |",
        "| 数据变换 | 降低影响 | 可解释性差 | 非正态分布 |",
        "| 分箱 | 鲁棒性强 | 损失信息 | 模型需要 |",
        "",
        "### 关键要点",
        "",
        "1. **先可视化后统计**：使用可视化方法了解异常值分布",
        "2. **结合业务理解**：判断异常值是否合理",
        "3. **选择合适方法**：根据数据分布选择检测方法",
        "4. **谨慎删除**：确认是错误数据才删除",
        "5. **记录处理过程**：标记异常值位置，保留信息",
        "6. **验证处理效果**：处理前后对比，确保合理",
        "7. **考虑模型类型**：线性模型必须处理，树模型可以保留"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.8.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}